DOCUMENT RESUME 



ED 343 370 



EC 301 056 



TITLE 

INSTITUTION 

SPONS AGENCY 

PUB DATE 
NOTE 
PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



An Analysis of the QEI Evaluation system. 
Research for Better Schools, Inc., Philadelphia, 
Pa. 

Pennsylvania State Dept. of Education, Harrisburg. 
Bureau of special Education. 
24 Jan 92 
33p. 

Reports - Evaluative/Feasibility (142) 
MF01/PC02 Plus Postage. 

•Disabilities; Elementary secondary Education; 
•Evaluation Methods? Formative Evaluation? Models? 
Outcomes of Education? Preschool Education? •Program 
Evaluation? Recordkeeping? state Programs? state 
Standards? Suminative Evaluation 
•Quality Education Initiative PA 



ABSTRACT 

This report evaluates the Quality Education 
Initiative (QEI) evaluation system developed m Pennsylvania to 
monitor and report progress of individual programs or sites serving 
students with and without disabilities. The system has three parts: a 
Program Documentation outline (PDO) , a Program Reporting outline 
(P5tO), and a Program Improvement outline (PIO) . This report 
summarizes analyses conducted of the PDOs, pros, and PlOs submitted 
by 21 QEI programs representing 7 different models. The study 
examined, first, the progress made by QEI programs in using the 
evaluation system and, second, the reported evaluation data. Findings 
indicated better performance with outcome than process evaluations. 
Programs reported achieving two-thirds of process objectives and 
two-fifths of outcome objectives. Recommendations address: (1) 
identification of key process and outcome evaluation objectives by 
the Pennsylvania Department of Education; (2) provision of training 
to QEI program personnel on how to evaluate key process and outcome 
objectives; (3) development of a common pool of instruments to assess 
program impacts on students, teachers, and parents; and (4) 
examination of the evaluations of the QEI programs as part of ongoing 
routine monitoring. QEI evaluation system forms and details of rating 
results are appended. (DB) 
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Executive Summary 



In 1989-90, the Pennsylvania Department of Education (PDE) contracted 
with Research for Better Schools (RBS) to develop an evaluation system as part 
of its Quality Education Initiative (QEI) . PDE intended the evaluation system 
to be used by individual programs or sites (i.e., school districts or 
intermediate units) to monitor and report their progress. In addition, PDE 
expected to use the resulting findings to evaluate the overall initiative and 
identify key QEI program elements. 

The OEI evaluation system has three parts -- a Program Documentation 
Outline (PDO), a Program Reporting Outline (PRO), and a Program Improvement 
Outline (PIO). The PDO is completed at the start of each year and lays out 
the program's objectives, measures (or instruments) for monitoring progress on 
these objectives, a timeline, and expectations (or levels) of performance 
anticipated for each objective. PDOs include both process and outcome 
evaluation objectives. PROs are completed at the end of the year and indicate 
the progress made by the program for each objective included on the PDO. 
Space is also provided to explain why an objective was accomplished or not. 
The PIO is filled in once all of the data for the PRO are analyzed and 
discrepancies known between the intended and actual performance of the 
program; the PIO describes what improvements will be made so that the program 
achieves the objectives in the future. 

Fifty- seven of the 104 QEI-funded programs attended training on this 
evaluation system during the first year. Following this initial training, 
these programs were expected to complete PDOs, PROs, and PIOs at the end of 
each year. This report summarizes analyses conducted of the PDOs, PROs, and 
PIOs submitted by 21 of the 57 QEI programs (36.8 percent) at the end of the 
second year. These 21 represented seven different models funded under QEI. 
Two sets of analyses were conducted? the first to determine the progress made 
by QEI programs in using the evaluation system and the second to summarize the 
evaluation data reported by the individual QEI programs. The findings from 
both sets of analyses were then used to make recommendations for the continued 
use of the QEI evaluation system in Pennsylvania. 

QEI Evaluation System 

The following three questions were used to focus the first set of 
analyses : 

• Were high quality process evaluations designed and conducted by 
individual programs 7 

• Were high quality outcome evaluations designed and conducted by 
individual programs? 

• Were evaluation findings used to identify improvements needed in QEI 
programs? 

The answers to each question are summarized below. 



Process evaluations . The process evaluations designed by the 21 
individual QEI programs were rated on a series of items. Individual items 



rated the quality of the process evaluation designed and carried out for the 
different key program elements that should be addressed as part of a high 
quality process evaluation {i.e., program planning, staff training, student 
selection and/or placement v implementation of key program components, 
monitoring of program implementation, and adoption of the model in other 
locations). High ratings indicated that specific objectives were set and 
substantive data presented for each key progr arena tic component, low ratings 
indicated the reverse. Analyses of these ratings revealed that the majority 
of QEI programs {17 of 21, or 81.0 percent) did not establish evaluation 
objectives for at least half of the six program elements. When objectives 
were established for any of the six program elements, the process evaluations 
generally relied on non-specific objectives and impressionistic data (47 out 
of 61 objectives, or 77.0 percent). Specific objectives were set only related 
to student identification and placement by six programs and in all six of 
these cases, impressionistic data also were reported. 

Outcome evaluations . Fourteen generic areas were identified by the 21 
QEI programs for potential impact. Most of these focused on student outcomes, 
including achievement, developmental status, attendance, failure or retention 
rate, receipt of instruction in regular classrooms, and referral to special 
education or Chapter 1 programs* The quality of the evaluations designed for 
each identified outcome were rated on a scale similar to that used to judge 
the process evaluations discussed above. The outcome evaluations almost 
always included specific objectives (52 out of 56 objectives, or 92.9 
percent). Of the 52, 15 reported substantivt data, 22 either relied on 
impressionistic data or reported no substantive data in impressionistic terms, 
and 15 included no data at the time the PDOs and PROs were submitted (because 
collected data had not yet been analyzed). QEI evaluations tended to report 
substantive data on outcomes which could be documented with fairly simple 
counts, such as student failure or retention rates, student referrals to 
special education or Chapter 1, and student placement in regular classrooms 
for academic instruction. Not surprisingly , QEI evaluation that emphasized 
outcome areas more difficult to measure tended to rely on impressionistic data 
or unvalidated measures to report on student, parent, or teacher progress. 

Program improvements based on evaluation findinRS . The third component 
of the evaluation system asked QEI programs to identify improvements that they 
would make based on the process and outcome evaluation findings reported in 
the PDOs and PROs. A rating scale was used to examine these improvements, a 
high score meant that program improvements were identified and their 
connections to evaluation findings were explicit. Almost one-fourth (23.8 
percent) of the programs were able to achieve some success in using evaluation 
data to identify necessary program improvements. Another 57,1 percent were 
able to identify needed changes, but did not link these changes to any 
evaluation data. Almost one- fifth (19.1 percent) were unable to identify 
changes for the coming year. 

Meta-Evaluation of QEI Findings 

To structure the second set of analyses, the following three questions 
were asked; 

# Are the QEI programs successfully implementing the various models? 



• Are the QEI programs achieving their intended outcomes for students, 
teachers, and parents? 

• What are the critical programmatic features in the QEI models for 
them to be successful? 

In order to answer these three questions, RBS relied entirely on the data 
presented by the 21 programs in their PDOs and PROs. As might be expected 
from the evaluation system results presented above, too often the program- 
supplied data were impressionistic or incomplete to allow for any additional 
analysis. Substantive data were reported for only 15 of the 56 possible 
student, teacher, or parent outcomes identified by QEI programs (26.8 
percent). In most of these cases, the data reported included pre- and 
post test measures for program students only? seldom were control or comparison 
data available. 

Given these circumstances, me ta -evaluation analyses to answer the above 
three questions seemed futile. RBS instead coded each objective as having 
been fully met, partially met, not met, or unknown. Approximately two-thirds 
of the QEI programs (66.2 percent) reported that they had accomplished all of 
their process evaluation objectives. Programs acknowledged some problems for 
about one-fifth of the process objectives, but never noted any serious 
problems that prohibited them from implementing the particular QEI model as 
intended. Slightly more than two-fifths (42.6 percent) of the programs 
reported complete success in achieving their outcome objectives and another 
18.7 percent noted partial success. Over one-third of the programs did not 
yet have their data and so the results for the QEI outcome evaluations may 
improve once missing data are analyzed. The final meta-evaluation question 
was not addressed given the paucity of data available. 

QEI Evaluation Findings and Recommendations 

The initial set of analyses indicate that process and outcome 
evaluations of various scope and quality are being carried out by QEI 
programs. When the ratings for the process and outcome evaluations are 
compared, it is clear that QEI programs are much more skilled at the latter. 
In general, they are able to develop specific evaluation objectives and 
measures to track program effects. This is especially the case in terms of 
student achievement, developmental status, attendance and referral, placement, 
and failure rates. They are less skilled in documenting student progress in 
other areas, or program effects on teachers or parents, often because they 
lack psychometrically sound measures to document progress. 

The QEI process evaluations tended to be less sophisticated than the 
outcome evaluations. For some programs, this may be due to a lack of 
understanding of how to conduct a high quality process evaluation. For other 
programs, process evaluations may be less valued because they may not provide 
any new information to small program staffs who know their programs extremely 
well. 

The abbreviated meta-evaluation analyses suggest that the Q£i programs 
are being implemented with some fidelity and are having some desirable impact 
on students, teachers, and parents. QEI programs reported that they achieved 
two-thirds of their process objectives and two-fifths of their outcome 
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objectives. These percentages are likely to increase once programs complete 
all of their analyses of program implementation and outcome data. In 
addition, the narrative reports submitted by QEI programs along with their 
PDOs, PROs, and PIOs were very complimentary. 

In spite of the positive impressions created by the individual programs' 
evaluations, more substantive data are necessary before the merits of QEI can 
be evaluated with any certainty or key features identified. In particular, 
substantive and comparable data are needed across some (or all) programs. 
Such data will enable analyses to establish the effects of a single program or 
model, the relative effectiveness of various models, and the identification of 
key progra mm atic elements. Until such data are collected and analyzed, it 
will be impossible for PDE to determine the success of QEI with any systematic 
certainty. 

In order for the current QEI evaluation system to meet any of these 
three needs, the following changes are recommended: 

• the key process and outcome evaluation objectives have to be 
identified by PDE 

• additional training should be provided to QEI programs on how to 
conduct evaluations on these key process and outcome evaluation 

objectives 

• a common pool of instruments should be developed to assess program 
impacts on students, teachers and/or parents 

• PDE should include examination of the evaluations of the QEI programs 
as part of their ongoing, routine monitoring. 
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Introduction 



In 1989-90, the Pennsylvania Department of Education (PDE) contracted 
with Research for Better Schools (RBS) to develop an evaluation system as part 
of its Quality Education Initiative (QEI). PDE intended the evaluation system 
to be used by school districts or intermediate units (IUs) to monitor and 
report their progress. In addition, PDE expected to use the resulting 
findings to evaluate the overall initiative and identify key QEI program 
elements . 

The QEI evaluation system has three parts -- a Program Documentation 
Outline (PDO), a Program Reporting Outline (PRO), and a Program Improvement 
Outline (PIO) . Copies of these three forms are included in Appendix A. The 
PDO is completed at the start of each year and lays out the program's 
objectives, measures (or instruments) for monitoring progress on these 
objectives, a timeline, and expectations (or levels) of performance 
anticipated for each objective. PDOs include both process and outcome 
evaluation objectives. PROs are completed at the end of the year and indicate 
the progress made by the program for each objective included on the PDO. 
Space is also provided to explain why an objective was accomplished or not. 
The PIO is filled in once all of the data for the PRO are analyzed and 
discrepancies known between the intended and actual performance of the 
program; the PIO describes what improvements will be made so that the program 
achieves the objectives in the future. 

As part of the QEI, 104 programs or sites (i.e., school districts or 
intermediate units) were funded in 1989-90. These 104 programs involved 14 
different models for addressing special needs of students. During the first 
year of this effort, staffs from 57 QEI programs were trained by RBS staff on 
the use of the evaluation system. Following this initial training, staff have 
completed the cycle twice, each time completing PDOs, PROs, and PIOs. These 
forms are submitted to PDE at the end of each funding year. 

This report is intended to serve three purposes. First, it describes 
the progress made by QEI programs in using the evaluation system. Particular 
attention is given to their understanding of the three forms and the quality 
of the evaluations designed by QEI program staff. Second, thp evaluation data 
reported by the individual programs are summarized and a meta-evaluation of 
the 1990-91 QEI programs is presented. Third, the report provides overall 
findings and recommendations related to the continued use of the QEI 
evaluation system in Pennsylvania. 



QEI Evaluation System 



This section of the report focuses on the use of the evaluation system 
by 21 QEI programs who submitted their evaluation system forms at the end of 
1990-91. These programs are listed below. 

• Appalachia Intermediate Unit 8 Project MELD (Mainstream Experience 
for Learning Disabled) 



• Berks County Intermediate Unit 14 Project MELD 

• Spring Cove School District Project MELD 

• School District of Philadelphia Project SEED 

• School District of Philadelphia Valentine Behavior Disorder 
Discipline Model 

• Daniel Boone Area School District ODDM (Outcomes-Driven Developmental 
Model) 

• South Side Area School District ODDM 

• Montgomery County Intermediate Unit CBAID (Curriculum-Based 
Assessment for Instruction Design) 

• ARIN Intermediate Unit 28 CBAID 

• School District of Philadelphia CBAID 

• Rivervi«»w School District Home School Liaison Project 

• BLaST Intermediate Unit 17 Preschool Integration Program 

• Keystone Central School District Preschool Integration Program 

• School District of Lancaster Preschool Integration Program 

• Danville Area School District Preschool Integration Program 

• Middletovm Area School District Preschool Integration Program 

• Commodore Perry School District Preschool Integration Program 

• Tuscarora Intermediate Unit 11 Preschool Integration Program 

• Farrell Area School District Preschool Integration Program 

• Hollidaysburg Area School District Preschool Integration Program 

• Steelton-Highspire School District Preschool Integration Program 

These 21 represent seven of the 14 QEI models funded by PDE. Another six 
programs submitted incomplete evaluation information and so were excluded from 
these analyse?. Thirty programs did not respond to requests from PDE for 
their evaluation results. 

In order to assess QEI programs' understanding and use of the evaluation 
system, the 1990-91 PDOs, pROs, and PlOs were carefully reviewed, along v 1th 
any other supplementary documentation provided by each program. In most cases 
this included a short, one or two page narrative, sample program information 
forms, and sometimes sample student records or test results. The unit of 



analysis thus focused on the individual program, or site, not the particular 
QEI model being implemented. The analyses focused on three questions: 

• Were high quality process evaluations designed and conducted by 
individual programs? 

• Were high quality outcome evaluations designed and conducted by 
individual programs? 

• Were evaluation findings used to identify improvements needed in QEI 
programs? 

Rating scales were devised to answer each of the above questions. The rating 
scales and resulting ratings are presented below. 

Process Evaluations 

Process evaluations are conducted primarily to determine whether 
programs are implemented as planned. Process evaluations of QEI programs 
should consequently focus on such key program elements as program planning 
(for implementation), staff training, student selection and/or placement, 
implementation of key program components, monitoring of program 
implementation, and adoption of the model in other IU/district locations. The 
PDOs and PROs for the 21 QEI programs were examined to determine the quality 
of the process evaluations proposed and conducted for each of these major 
program elements. Particular attention was paid to the delineation of 
specific objective (s) for each program element as well as the collection, 
analysis, and reporting of substantive data on each element. The process 
evaluation carried out for these six program elements, as presented in the 
program's PDO and PRO. was rated or. the following six-point scale: 

1 no objective s»?t, no data presented 

2 non-specific objective set, no data presented 

3 non-specific objective set, impressionistic data presented 

4 specific objective set, no data presented 

5 specific objective set, impressionistic data presented 

6 specific objective set, substantive data presented. 

An objective was considered specific if it referred to a particular feature of 
the program (e.g., the selection of development ally delayed children using the 
Batelle Developmental Inventory), if the objective called for the 
identification of students without naming the instrument or selection process 
it was considered a non-specific objective. Similarly, evaluation data and 
results were considered to be substantive if the measure wati named and the 
data wt-re defined in explicit terms (e.g., 95 percent of the targeted children 
made 8 or more months progress on the Batelle Developmental Inventory). If 
the evaluation results were less specific (e.g., the teachers reported that 
most of the students made gains), they were classified as impressionistic 
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The aggregated process evaluation ratings are presented in Table 1 
below. (See Appendix B for ratings for the individual QEI programs.) The 
analyses of process evaluation quality revealed that the majority of QEI 
programs (17 of 21 9 or 81.0 percent) did not establish evaluation objectives 
for at least half of the above six program elements. The majority did not set 
objectives for program planning, monitoring of program implementation, or 
adoption of the model in other locations. In contrast, most QEI programs set 
objectives for staff training, student identification and/or placement, and 
implementation of key program components. When objectives were established 
for any of the six program elements, the process evaluations generally relied 
on non-specific objectives and impressionistic data (47 out of 61 objectives, 
77.0 percent). This was especially the case in evaluating staff training (15 
out of 16 programs) and imple mentation of key program components (13 out of 16 
QEI programs). Specific objectives were set only related to student 
identification and placement by six programs and in all six of these cases, 
impressionistic data also were reported. 

Outcome Evaluations 

Outcome evaluations are conducted to determine whether programs achieve 
the expected results* QEI outcome evaluations should accordingly focus on 
students who participate in or receive program services. In addition, some 
QEI programs expected to have an impact on the teachers or parents of students 
participating in the program. Fourteen generic areas were identified by the 
21 QEI programs for potential impacts 



• 


student 


achievement 


• 


student 


developmental status 


• 


student 


attendance 


• 


student 


failure or retention rate 


• 


student 


behavior or discipline 


• 


student 


self-esteem or affective/emotional development 


• 


student 


interpersonal skills 


• 


student 


self-directed learning 


• 


student 


completion of homework 


» 


student 


receipt of instruction in regular classroom 


• 


student 


referral to special education/Chapter 1 


• 


parent involvement 


• 


teacher perception of student 


• 


teacher 


skills. 




Table 1 

Quality of Process Evaluation Ratings 











Staff 




Student 


Implementation of 


Monitoring 






Ratinfi 


Planning 


Training 




ID 


Key Elements 


Implements t ion 


Adoption 


N 


Percent 


N 


Percent 


N 


Percent 


N 


Percent 


N 


Percent 


N 


Percent 


1 


16 


76.2 


5 


23.8 


10 


47.6 


5 


23.8 


13 


61.9 


16 


76.2 


2 


1 


4.8 


1 


4.8 


1 


4.8 


3 


14.3 


2 


9.5 


0 


0.0 


3 


4 


19.0 


15 


71.4 


4 


19.0 


13 


61.9 


6 


28.6 


5 


23.8 


4 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 


5 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 


6 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 


0 


0.0 



Note: Ratings were assigned on the following six-point scale: 1-no objective set, no data presented, 2-non-specific 
objective set, no data presented, 3-non-specific objective s impressionistic data presented, 4-specific 
objective set, no data presented, 5-specific objective set, impressionistic data presented, 6-sp»cific objective 
set, substantive data presented ' 
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The number of outcomes identified by QEI programs ranged from one to five, 
with seven programs reporting one outcome, six programs identifying two or 
three, and eight programs four or five. The PDOs and PROs were examined to 
assess the quality of the outcome evaluations proposed and conducted for the 
particular set of outcomes identified by the project. Ratings were assigned 
for each outcome, using the same rating scale that was used in assessing the 
quality of the process evaluation. Similar to those analyses, the delineation 
of specific objectives and the collection, analysis, and reporting of 
substantive data were particularly important. 

The outcome evaluation ratings of quality are presented in Table 2 
below. (Appendix C contains the individual program ratings for the outcome 
evaluations.) The QEI outcome evaluations almost always included specific 
objectives (52 out of 56 chjectives, or 92.9 percent). Of the 52, 15 reported 
substantive data, 22 either relied on impressionistic data or reported 
substantive data in impressionistic terms, and 15 included no data at the time 
the PDOs and PROs were submitted (because the data analyses had not yet been 
completed). In marked contrast to the process evaluations, the QEI outcome 
evaluations were more focused and sophisticated in their design. 

The majority of the QEI outcome evaluations focused on tracking student 
progress. All but two of the 21 QEI programs identified either improved 
student achievement or developomental status as a potential impact area; 18 of 
these 19 also identified specific objectives and measures to track student 
progress. QEI evaluations tended to report substantive data on outcomes which 
could be documented with fairly simple counts, such as student failure or 
retention rates, student referrals to special education or Chapter 1, and 
student placement in regular classrooms for academic instruction. Not 
surprisingly, QEI evaluations that emphasized outcome areas more difficult to 
measure by simple counts, such as student affective development, parent 
involvement or teacher attitudes or skills, tended to rely on impressionistic 
data or unvalidated measures to report on student, parent, or teacher 
progress . 

Program Improvements Based on Evaluation Findings 

The third component of the evaluation system asked QEI programs to 
identify improvements that they would make based on the process and outcome 
evaluation findings reported in the PDOs and PROs . The PDOs, PROs and PIOb 
were examined together to determine whether individual programs were able to 
complete the task. The following four-point rating scale was developed to 
structure this examination! 

1 program improvements not identified 

2 program improvements identified, no connection made to evaluation 
findings 

3 program improvements identified, connections to evaluation findings 
sometimes explicit B 

4 ^^improvements identified, connections to evaluating findings 
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Table 2 

Quality of Outcome Evaluation Ratings 



Riling Achievement 



1 
2 
3 
4 
5 
6 



0 
0 
0 
5 
3 
1 



0.0 
0.0 
0.0 
55.6 
33.3 
11.1 



Tola! 0 100.0 



Development 
Status 



Attendance Failure Rata Behavior 



N Percent N Percent N Portent N Portent H 



9 
0 
1 
2 
3 
4 



0.0 
0.0 
10.0 
20.0 
30.0 
40.0 



10 100.0 



0 
0 
0 
2 
2 



0.0 
0*0 
0.0 
50.0 
50.0 
0.0 



4 100.0 



0.0 
0.0 
0.0 
20.0 
20,0 
60.0 



Percent 

0.0 
0.0 
0.0 
0.0 
100.0 
0.0 



5 100.0 2 100.0 



Affective 


Inter 


persona! 


Se!f-Dlretted 


Development 




Mils 


learning 


H Portent 


N Percent 


H Percent 


0 0.0 


0 


0.0 


0 0.0 


0 0.0 


0 


0.0 


0 0.0 


0 0.0 


1 


25.0 


0 0.0 


1 50.0 


0 


0.0 


0 0.0 


1 50.0 


3 


75.0 


1 100.0 


0 0.0 


0 


0.0 


0 0J 


2 100.0 


4 


100.0 


1 100.0 



Homework 


Instruction In 


Referred to 


Parent 


Teacher 






Regular Class 


Spec. fd/tn. 1 


Involvement 


Porteptloai 


H 


Percent 


N Portent 


N Portent 


N Portent 


N Per*** 


0 


0.0 


0 0.0 


0 0.0 


0 0.0 


0 0.0 


0 


0.0 


O 0.0 


0 0.0 


0 O.0 


0 0.0 


0 


0.0 


0 0.0 


0 0.0 


0 0.0 


1 33.3 


0 


0.0 


0 0.0 


2 28.6 


1 33.3 


1 33.3 


1 


100.0 


0 0.0 


1 14.3 


2 56,7 


1 33.3 


0 


0.0 


2 100.0 


4 57,1 


0 0.0 


O 0.0 


1 


100 .0 


2 100.0 


7 1OO.0 


3 100.0 


3 100.0 



Teathor 

Mllte 

N Portent 



0.0 
0.0 
33.3 
0.0 
33.3 
33.3 



Total 



4 

15 
22 
15 



3 100.0 56 



iSElB^^ 2* <hi » ^lS h * D . tal w i'? : 1 * n0 presented. 2*oo.teeclffe objective set, no data presented, 3-non-speetffc ob|octlvo eel, 

impress lenlstfc data presented, 4-ipoclflc objective set, no data presented. 5* pacific objective eel. Impressionistic dale presented, 6-tpeellfc objective eet, subsrsntlve data presented 
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Table 3 below summarizes the resulting ratings for the 21 QEI programs (see 
Appendix D for the individual program ratings). 



Table 3 



Between 



Ratings of Connections Made 
Evaluation Findings and Improvements 



Rating 



Number 



Percent 



1 
2 
3 
4 



4 
12 
5 
0 



19.1 
57.1 
23.8 
0.0 



Almost one-fourth (23.8 percent) of the programs were able to achieve some 
success in using evaluation data to identify necessary program improvements. 
Another 57,1 percent were able to identify needed changes, but did not link 
these changes to any evaluation data. Finally , almost one-fifth were unable 
to identify changes for the coming year. 



The second purpose of this report is to provide feedback to PDE on the 
success of QEI programs. This section of the report is thus organized to 
answer three questions: 

• Are the QEI programs successfully implementing the various models? 

• Are the QEI programs achieving their intended outcomes for students, 
teachers, and parents? 

• What are the critical programmatic features in the QEI models for 
them to be successful? 

In order to answer these three questions, RBS relied on the evaluation 
findings presented oy the 21 programs listed at the beginning of the report. 
Our answers to these questions thus depend entirely on the validity and 
completeness of the data presented by the? programs in their PDOs and PROs. As 
might be expected from the evaluation system results presented in the first 
section of this report, too often the program-supplied data were 
impressionistic (e.g., the teachers noted that students made marked gains in 
reading during the course of the school year) or incomplete to allow for any 
Stteta~anal' sis . Substantive data were reported for only 15 out of the 56 
possible student, teacher, or parent outcomes identified by QEI programs (26.8 
percent). Jn most of these cases, the data reported included pre- and 
posttest measures for program students only; seldom were control or comparison 
student data available. 

Given these circumstances. r«eta-analyses to answer the above three 
questions seemed futile. RBS instead decided to code each objective listed in 
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a program's PDO and FRO (both process and outcome) as having been fully met, 
partially met, not met, or unknown. (The unknown category was used when data 
analyses had not been completed and so achievement of the objective could not 
be determined.) RfiS relied entirely on the judgments made by projects 
themselves in classifying objectives as having been accomplished or not. 
These data are presented in Table 4 below. (The individual QEI program 
ratings are included in Appendix E.) 

Table 4 

Number of Process and Outcome Objectives 
Met by QEI Programs 



Process Outcome 



Objectives 


Number 


Percent 


Number 


Percent 


Fully Met 


100 


66.2 


32 


42.6 


Partially Met 


30 


19,9 


14 


18.7 


Not Met 


0 


0.0 


2 


2.7 


Unknown 


21 


13.9 


27 


36.0 


Total 


151 


100.0 


75 


100.0 



As indicated in Table 4, approximately two- thirds of the QEI programs reported 
that they had accomplished all of their process evaluation objectives. 
Programs acknowledged some problems for about one-fifth of the process 
objectives, but never noted any serious problems that prohibited them from 
implementing the particular QEI model as intended. 

On first glance, the results for the outcome evaluation objectives seem 
somewhat less optimistic. Slightly more than two-fifths (42.6 percent) of the 
programs reported complete success, in contrast to the two- thirds (66.2 
percent) noted for the process objectives. However, closer examination shows 
that results were not yet avilable for over one-third (36*0 percent) of the 
outcome objectives and so it is impossible to predict whether programs met 
these objectives or not. 

The final meta-evaluation question sought to identify the critical 
features necessary for QEI models to be successful. Given the paucity of data 
available, such identification seems premature. Until there is more 
substantive evidence to document the effectiveness of any of these programs, 
it is impossible to reliably determine which programmatic factors are 
essential to the effective implementation of QEI programs. 



QEI Evaluation FindinRs and Recommendations 



The initial set of analyses indicate that process and outcome 
evaluations of various scope and quality are being carried out by QEI 
programs. When the ratings for the process and outcome evaluations are 
compared, it is clear that QEI programs are much more skilled at the latter. 
In general, they are able to develop specific evaluation objectives and 
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measures to track program effects* This is especially the case in terms of 
student achievement, developmental status, attendance and referral, placement, 
and failure rates* They are less skilled in documenting student progress in 
other areas, or program effects on teachers or parents, often because they 
lack psychometrically sound measures to document progress. 

The QEI process evaluations tended to be less sophisticated than the 
outcome evaluations. For some programs, this may be due to a lack of 
understanding of how to conduct a high quality process evaluation. For 
example, one common approach was to state one over-arching process evaluation 
objective, implement key elements of the program, without ever noting what the 
critical components were or reporting data on the implementation of these 
individual components. In other cases, programs adopting the same QEI model 
would often use the same structure for designing their process evaluation and 
submit essentially the same PDOs and PROs, thus failing to tailor the process 
evaluation to match the particular conditions or nuances at their individual 
schools . 

Another reason for the differences found between the process and outcome 
evaluations may be that the differences are an artifact of the relatively 
small size of many of these programs, QEI programs typically involved only a 
handful of professional staff in each school or district. In many of these 
programs, one can assume that the Individual responsible for the evaluation 
was often intimately involved in the program on a day-to-day basis and so knew 
first-hand the ups- and- downs of the program's implementation* Under these 
conditions, the process evaluation is not likely to shed any new light on the 
program, at least from the perspective of the program staff. The process 
evaluation, consequently, becomes less important, takes up valuable time 
needed elsewhere, ard is completed mainly to satisfy the funding agency. 

Given these two scenarios, it's not surprising to discover that QEI 
programs were more proficient at conducting outcome than process evaluations. 
However, the solutions for dealing with these two circumstances are very 
different. In the case of the former, additional training, or one-on-one 
coaching is probably needed. In the latter instance, additional training 
won't help, and may actually be resented. 

The abbreviated meta-analyses suggest that the QEI programs are being 
implemented with some fidelity and are having some desirable impact on 
students, teachers, and parents. QEI programs reported that they achieved 
two-thirds of their process objectives and two-fifths of their outcome 
objectives. These percentages are likely to increase once programs complete 
all of their analyses of program implementation and outcome data. In 
addition, the narrative reports submitted by QEI programs along with their 
PDOs, PROs, and PIOs were very complimentary. 

In spite of the positive impressions created by the individual programs* 
evaluations, more substantive data are necessary before the merits of QEI can 
be evaluated with any certainty or key features identified. In particular, 
substantive and comparable data are needed across some (or all) programs. 
Such data will enable analyses to establish the effects of a single program or 
model, the relative effectiveness of various models, and the identification of 
key programmatic elements. Until such data are collected and analyzed, it 
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will be impossible for FDE to determine the success of QEI with any systematic 
certainty. 

In order for the current QEI evaluation system to meet any of these 
three needs, the following changes are recommended, 

• The key process and outcome evaluation objectives have to be 
identified by PDE. 

The current evaluation system allows individual QEI programs to decide 
on the specific process and outcome objectives their evaluations will address 
as long as the PDO-PR0-PI0 framework is used. Although this flexibility 
provides a strong incentive to programs to conduct their evaluations, it also 
works against PDE's interest of gathering some uniform data so that models and 
programs can be studied as a group. RBS therefore suggests that some common 
process and outcome objectives be required so that the state's interests are 
also served. 

• Additional training should be provide to QEI programs on how to 
conduct evaluations on these key process and outcome evaluation 
objectives. 

In reviewing the PDOs-PROs-PIOs submitted by the programs f it was clear 
that few of them had mastered the evaluation system designed for QEI. 
Therefore, it seems wise for PDE to provided additional training to these 
programs, and to focus this training on those elements of the evaluation 
system most critical to PDE and QEI* Some large group training may be 
appropriate to make sure that all of the programs share a common focus and 
understanding of the QEI evaluation system. However, given the variance in 
the evaluation skill levels of the individual program staffs, some of the 
training ougnt to be tailored to meet different skill levels. This could be 
accomplished in small group training sessions or individual coaching. In 
either case, the additional follow-up training ought to build on the 
evaluations designed by these programs during the previous years (e.g., 1989- 
90 t 1990-91) so that the evaluations are on-target this school year. 

• A common pool of instruments should be developed to assess program 
impacts on students, teachers^ and/or parents. 

If meta-analyses are to be conducted to examine the relative success of 
individual QEI models or programs, it would be extremely helpful if similar 
data were collected across programs. This is especially the case for student 
outcomes. RBS recommends that PDE encourage individual QEI programs striving 
for similar student outcomes to discuss and share instruments, and perhaps 
reach some agreement on which instruments to use to assess particular 
outcomes. Without some formal or informal understandings among individual 
programs, any meta-analyses of relative effects will be compromised. Such 
sharings may also have the by-product of helping some programs identify more 
psychometrically sound measures for assessing particular outcomes. 

• PDE should include examination of the evaluations of the QEI programs 
as part of their ongoing, routine monitoring. 
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During the year, PDE officials make periodic monitoring visits to QEI 
programs to review and discuss program operations with their staff members. 
During these monitoring visits, PDE officials should extend their attention 
beyond program operations and services to include the status of evaluation 
activities and preliminary results. By engaging local district/school or IU 
staff in discussions about the evaluation on an ongoing basis, PDE will 
communicate its commitment «.o and the importance attached to the QEI 
evaluation effort. This, in turn, will promote the evaluation activities 
carried out by district /school or IU staff. 
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QEI Evaluation System Forms 
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PROGRAM DOCUMENTATION OUTLINE 



PAGE OF 



NAME OF PROGRAM: 



REVISED: 



(A) 

OBJECTIVES 



(B) 

MEASURES 
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erIc 



(C) 
TIME 



(D) 

EXPECTATIONS 
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PROGRAM REPORTING OUTLINE 



NAME OF PROGRAM: 



REVISED: 



(A) 

OBJECTIVES 
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0 



(E) 

ACCOMPLISHMENTS 



(F) 

EXPLANATION 
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PROGRAM IMPROVEMENT OUTLINE 



NAME OF PROGRAM: 



REVISED: 



(A) 

OBJECTIVES 



IMPROVEMENTS 



(H) 

RATIONALE 
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Appendix B 

Ratings of Process Evaluation Objectives 
for QEI Programs 



Imp 1 em. of 







Staff 


Student 


Program 


Monitoring 


Adopt 




PI Ann ins 


Training 


ID 


Elements 


Implem. 




1 


3 


3 


1 


3 


1 


UU £ 


1 


3 


3 


1 


3 


1 




1 


3 


3 


1 


3 


1 


UvJ 




3 


1 


3 


1 


1 


uuo 




3 


1 


3 


3 


3 


nofl 
uuo 


1 


1 


1 


1 


1 


3 


000 


1 


3 


1 


2 


2 


1 


010 


1 


3 


1 


3 


1 


3 


Oil 


3 


3 


3 


3 


1 


3 


012 


3 


3 


1 


3 


1 


1 


013 


1 


1 


1 


1 


1 


1 


014 


1 


3 


5 


3 


3 


1 


015 


1 


1 


5 


2 


1 


1 


016 


1 


3 


5 


3 


1 


1 


017 


1 


3 


5 


3 


3 


3 


018 


1 


3 


5 


3 


1 


1 


019 


1 


1 


2 


2 


1 


1 


.20 


1 


3 


5 


3 


1 


1 


021 


1 


1 


1 


3 


1 


1 


022 


1 


2 


1 


3 


2 


1 


023 


2 


3 


1 


3 


1 


1 



Note: Ratings were assigned on the following six-point scales 1-no objective set, no 
data present, 2-non-speclf ic objective set, no data presented, 3-non-specific 
objective set, impressionistic data presented, 4-specific objective set, no data 
presented, 5-specific objective set, impressionistic data presented, 6-specific 
objective set, substantive data presented 
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Appendix C 

Ratings of Outcome Evaluation Objectives for QEI Programs 



Dove! 

Project AebieramiM SW 



861 



Attentlince 



Bite 



Affective 
Development 



fcteroorvonsl 
Skills 



Salf-Okvdsd 
Ltimtad 



Homework 



Imtntetioelo 



894 
885 



889 
818 
811 
812 
813 
814 
815 
61S 
817 
018 
819 
628 
& 621 
00 822 



5 
1 



Rafamd to 
Sow. Etf/Cb. 1 

4 

8 



Pirtnl 
limlrannflt 



Teacher 
Perceptions 



SUItt 



Note; fUliiigt m tssigied on the foHowtei tti-potat scale: 1-ao objtcthn set, 00 tat presented, 2-flOMperifte objective ut, eo date p restated, S-«on -specific ebtetiive tit, 
impressionistic data pnisanied, Specific objective set, 00 data presented, S-ipeclfic objective set, impressionists Ma presented, fr-sptrific objective set, substantive sets presented 
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Appendix P 

Ratings of Connections Made Between Evaluation Findings 
and Improvement for QEI Programs 



Project Rating 



001 


i 


002 


•> 

z 


004 


2 


005 


1 


006 


3 


008 


1 


009 


2 


010 


3 


011 


2 


012 


3 


013 


3 


014 


2 


015 


■> 


016 


1 


017 


2 


018 


2 


019 


2 


020 


1 


021 


2 


022 


3 


023 


2 



Note: Ratings were assigned on the following four-point scale: 1-program 
improvements not identified, 2- program improvements identified, no 
connection to evaluation findings made, 3-program improvements 
identified, connections to evaluation findings sometimes explicit, 
4-program improvements identified, connections to evaluation findings 
explicit 
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Appendix £ 

Number of Process and Outcome Objectives 
Met by QEI Objectives 



Process Objectives Outcome Objectives 



Program 


Not 


Partially 


Fully 




Not 


Partially 


Fully 




Number 


Met 


Met 


Met 


Unknown 


Met 


Met 


Met 


Unknown 


001 


0 


2 


4 


0 


0 


0 


0 


4 


002 


0 


0 


3 


3 


1 


0 


2 


1 


003 


0 


0 


6 


0 


0 


0 


3 


1 


005 


0 


0 


5 


0 


0 


0 


0 


1 


006 


0 


1 


4 


0 


0 


1 


1 


0 


008 


0 


0 


2 


0 


0 


0 


it 


0 


009 


0 


2 


2 


0 


1 


0 


2 


4 


010 


0 


10 


10 


0 


0 


3 


2 


0 


Oil 


0 


6 


18 


0 


0 


0 


2 


0 


012 


0 


0 


3 


0 


0 


0 


4 


0 


013 


0 


0 


0 


0 


0 


6 


0 


0 


014 


0 


0 


2 


2 


0 


1 


2 


0 


015 


0 


0 


4 


0 


0 


0 


2 


0 


016 


0 


2 


5 


0 


0 


0 


1 


0 


017 


0 


2 


4 


0 


0 


0 


2 


0 


018 


0 


0 


5 


0 


0 


1 


2 


0 


019 


0 


0 


0 


14 


0 


0 


0 


8 


020 


0 


2 


4 


0 


0 


0 


1 


0 


021 


0 


1 


10 


0 


0 


0 


0 


8 


022 


0 


2 




2 


0 


2 


1 


0 


023 


0 


0 


5 


0 


0 


0 


1 


0 
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